Reading Assignment 2

SVM soft-margins, overfitting, training and validation sets, kernel SVM

Write your answers in a PDF and upload the document on Gradescope for submission. The due date is given on Gradescope.

Each question is worth 10 points.

Please watch the videos and slides before answering these questions.

  1. 1.7 Soft-margin. Explain what soft margins are.
  2. 1.7 Soft-margin. Give an example where soft-margins are required.
  3. 1.7 Soft-margin. Write the optimization problem with soft-margins.
  4. 1.7 Soft-margin. What happens when $\xi_i > 1$?
  5. 1.8 Overfitting. Explain what happens when we increase or decrease C.
  6. 1.9 Training and validation sets. Give an example of hyper-parameter for SVM.
  7. 1.9 Training and validation sets. Explain how hyper-parameters are optimized using validation sets.
  8. 1.9 Training and validation sets. What problem does cross-validation address?
  9. 1.9 Training and validation sets. Explain the difference between a validation set and a test set.
  10. 1.10 Kernel trick. What is the motivation for using kernels in SVM, as opposed to using raw data coordinates directly?
  11. 1.10 Kernel trick. Provide some definition of $\phi(x)$ that is non-linear. Describe the family of decision curves that can be described using SVM with $\phi(x)$.
  12. 1.10 Kernel trick. Assuming we are not using soft-margins, what are the possible values of $\sum_i \alpha_i K(x^{(i)},x) + b$ when $x$ is a support vector?
  13. 1.10 Kernel trick. Explain the role of $\gamma$ in the RBF kernel.
  14. 1.10 Kernel trick. Describe an example case where an RBF kernel will perform better than a linear kernel.